Source scheduler failure metric labels from error categorization, remove TrackedErrorRegexes#4980
Merged
dejanzele merged 2 commits intoJun 26, 2026
Conversation
dejanzele
force-pushed
the
rewire-scheduler-error-metric
branch
from
5047645 to
5eeb998
Compare
Contributor
Greptile SummaryThis PR replaces regex-based error classification with direct reads from
Confidence Score: 5/5Safe to merge — the change is a straightforward removal of dead regex machinery in favour of reading two proto fields that are already populated by the executor. The only observable side-effect is a change in metric label values (e.g. podError to user_error), which is explicitly documented. The path from failure detection to counter increment is now a single getter call on a nil-safe proto receiver, and call sites in schedulerapp.go and tests are consistently updated. No new error paths are introduced. No files require special attention. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Job Run Fails] --> B[ReportStateTransitions]
B --> C{jst.Failed?}
C -- Yes --> D[Lookup jobRunError by run ID]
D --> E[Read FailureCategory and FailureSubcategory from proto]
E --> F[Increment jobErrorsByQueue counter]
E --> G[Increment jobErrorsByNode counter]
subgraph OLD ["Removed - TrackedErrorRegexes path"]
H[errorTypeAndMessageFromError] --> I[Loop over compiled regexes]
I --> J[Return type and first matched regex string]
end
style OLD fill:#ffcccc,stroke:#cc0000
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
A[Job Run Fails] --> B[ReportStateTransitions]
B --> C{jst.Failed?}
C -- Yes --> D[Lookup jobRunError by run ID]
D --> E[Read FailureCategory and FailureSubcategory from proto]
E --> F[Increment jobErrorsByQueue counter]
E --> G[Increment jobErrorsByNode counter]
subgraph OLD ["Removed - TrackedErrorRegexes path"]
H[errorTypeAndMessageFromError] --> I[Loop over compiled regexes]
I --> J[Return type and first matched regex string]
end
style OLD fill:#ffcccc,stroke:#cc0000
Reviews (11): Last reviewed commit: "Merge branch 'master' into rewire-schedu..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
dejanzele
force-pushed
the
rewire-scheduler-error-metric
branch
from
5eeb998 to
91177c3
Compare
dejanzele
force-pushed
the
rewire-scheduler-error-metric
branch
7 times, most recently
from
4e2715f to
b0dd7f9
Compare
…lureSubcategory and remove TrackedErrorRegexes Signed-off-by: Dejan Zele Pejchev <pejcev.dejan@gmail.com>
dejanzele
force-pushed
the
rewire-scheduler-error-metric
branch
from
b0dd7f9 to
3af6b9f
Compare
JamesMurkin
approved these changes
Jun 26, 2026
Contributor
|
Tick the box to add this pull request to the merge queue (same as
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Armada exposes now
armada_scheduler_job_error_classification_by_queueand_by_nodenow label failures with the semantic category from error categorization, read off theErrorproto (FailureCategory/FailureSubcategory) instead of a regex match against the message. The metric names and label sets are unchanged. Only the label values change.categorywas theError.Reasontype (podError,leaseExpired, ...) and is now the semantic category, so dashboards filtering on the old values need updating.subcategorywas the first matching regex (empty in practice) and is nowFailureSubcategory.This replaces
trackedErrorRegexes, which is removed along with it: thescheduler.metrics.trackedErrorRegexesconfig and theerrorTypeAndMessageFromError/errorRegexesplumbing inmetrics.New. It was never set in-repo, so a deployment still setting it just logs an "unused key" warning.Validation
End to end on a Helm-deployed stack: failing jobs classified by the executor (
user_error,oom) and a killed-executor run (internal/lease-expired) all landed on the metric with the expected labels and counts.max-runs-exceededandjob-rejectedare job-level rather than run-level errors, so they do not surface in this metric.The metric names and label sets are unchanged, so the existing PrometheusRule keeps evaluating as before. Pre-existing bugs in that file are fixed separately in #4983.